Conditional Random Fields

In this report, we investigate Conditional Random Fields (CRFs), a family of conditionally trained undirected graphical models. We give an overview of linear CRFs that correspond to chain-shaped models and show how the marginals, partition function and MAP-labelings can be computed. Then, we discuss various approaches for training such models ranging from the traditional method of maximizing the conditional likelihood or its variants like the pseudo likelihood to margin maximization. For the margin-based formulation, we look at two approaches the SMO algorithm and the exponentiated gradient algorithm. We also discuss two other training approaches one that attempts at removing the regularization term and other that uses a kind of boosting to train the model. Apart from training, we look at topics like the extension to segment level CRFs, inducing features for CRFs, scaling them to large label sets, and performing MAP inferencing in the presence of constraints. From linear CRFs, we move on to arbitrary CRFs and discuss exact algorithms for performing inferencing and the hardness of the problem. We go over a special class of models Associative Markov Networks, which are applicable in some real-life scenarios and which permit efficient inferencing. We then look at collective classification as an application of general undirected models. Finally, we very briefly summarize the work that could not be covered in this report and look at possible future directions. 1 Undirected Graphical Models Let X = X1, . . . , Xn be a set of n random variables. Assume that p(X) is a joint probability distribution over these random variables. Let XA and XB be two subsets of X which are known to be conditionally independent, given XC . Then, p(.) respects this conditional independence statement if p(XA|XB , XC) = p(XA|XC) (1) or alternatively, p(XA, XB |XC) = p(XA, XB , XC) p(XC) = p(XA|XB , XC)p(XB , XC) p(XC) = p(XA|XC)p(XB |XC) (2) The shorthand notation for such a statement is : XA ⊥ XB |XC . Given X and a list of such conditional independence statements, we would like to characterize the family of joint probability distributions over X that satisfy all these statements. To achieve this, consider an undirected graph G = (X,E) whose vertices correspond to our set of random variables. We would construct the edge set E in such a manner that the following property holds: If the deletion of all vertices in XC from the graph results in the removal of all paths from XA to XB , then XA ⊥ XB |XC . Conversely, given an undirected graph G = (X,E), we can exhaustively enumerate all conditional independence ∗[email protected]